Skip to content

VED-1017: Generate JSON ACK file#1168

Merged
JamesW1-NHS merged 49 commits intomasterfrom
VED-1017-json-ack-file
Feb 10, 2026
Merged

VED-1017: Generate JSON ACK file#1168
JamesW1-NHS merged 49 commits intomasterfrom
VED-1017-json-ack-file

Conversation

@JamesW1-NHS
Copy link
Contributor

@JamesW1-NHS JamesW1-NHS commented Jan 28, 2026 •
edited
Loading

Summary

PR #1177 with the e2e_automation test changes merged manually. This one (not #1177) is to be merged to master.

The PRs are identical, except that #1168 has no unverified commits.
For confidence, you can verify that the branches are identical by fetching them locally and doing a diff i.e.
git fetch origin
git diff VED-1017-test-merge-e2e VED-1017-json-ack-file

  • Routine Change
  • ❗ Breaking Change

Acceptance Criteria

A structured JSON ACK is generated when batch processing completes (EOF received / completion confirmed).

The ACK includes:

  • Batch-level metadata (e.g. system, version, generated date, filename, provider)
  • A summary section showing total, successful, and failed record counts
  • A failures section listing only failed records with relevant error details
  • No per-record success acknowledgements are included
  • The ACK file is generated only once batch completion is confirmed (via EOF)

Conforms to the Design document: https://nhsd-confluence.digital.nhs.uk/pages/viewpage.action?pageId=1187373499&spaceKey=Vacc&title=Immunisation%2BFHIR%2BAPI%2B-%2BACK%2BFile%2BManagement#addcomment

Note: I have marked this as a breaking change as it requires changes to the E2E Automation Tests. These are ongoing. This currently builds successfully in master because the tests requiring modification have been temporarily patched.

Note to Testers

  • Changes are to ack_backend only. No change to API or other lambdas.
  • E2E Automation tests require modification (see above).

Reviews Required

  • Dev
  • Test
  • Tech Author
  • Product Owner

Review Checklist

ℹ️ This section is to be filled in by the reviewer.

  • I have reviewed the changes in this PR and they fill all of the acceptance criteria of the ticket.
  • If there were infrastructure, operational, or build changes, I have made sure there is sufficient evidence that the changes will work.
  • If there were changes that are outside of the regular release processes e.g. account infrastructure to setup, manual setup for external API integrations, secrets to set, then I have checked that the developer has flagged this to the Tech Lead as release steps.
  • I have checked that no Personal Identifiable Data (PID) is logged as part of the changes.

@JamesW1-NHS JamesW1-NHS marked this pull request as draft January 28, 2026 15:26
@github-actions
Copy link
Contributor

github-actions bot commented Jan 28, 2026

This branch is working on a ticket in the NHS England VED JIRA Project. Here's a handy link to the ticket:

VED-1017

@JamesW1-NHS JamesW1-NHS temporarily deployed to internal-dev-sandbox January 28, 2026 15:33 — with GitHub Actions Inactive
@JamesW1-NHS JamesW1-NHS temporarily deployed to internal-dev-sandbox January 28, 2026 15:37 — with GitHub Actions Inactive
@JamesW1-NHS JamesW1-NHS temporarily deployed to internal-dev-sandbox January 29, 2026 17:16 — with GitHub Actions Inactive
@JamesW1-NHS JamesW1-NHS temporarily deployed to internal-dev-sandbox January 29, 2026 17:20 — with GitHub Actions Inactive
@JamesW1-NHS JamesW1-NHS temporarily deployed to internal-dev-sandbox January 29, 2026 17:53 — with GitHub Actions Inactive
dlzhry2nhs
dlzhry2nhs previously approved these changes Feb 9, 2026
View reviewed changes
@JamesW1-NHS JamesW1-NHS marked this pull request as draft February 9, 2026 18:10
@JamesW1-NHS JamesW1-NHS temporarily deployed to internal-dev-sandbox February 10, 2026 09:43 — with GitHub Actions Inactive
@JamesW1-NHS JamesW1-NHS temporarily deployed to internal-dev-sandbox February 10, 2026 09:49 — with GitHub Actions Inactive
@JamesW1-NHS JamesW1-NHS temporarily deployed to internal-dev-sandbox February 10, 2026 09:49 — with GitHub Actions Inactive
@JamesW1-NHS JamesW1-NHS marked this pull request as ready for review February 10, 2026 11:05
@JamesW1-NHS JamesW1-NHS temporarily deployed to internal-dev-sandbox February 10, 2026 11:08 — with GitHub Actions Inactive
@sonarqubecloud
Copy link

sonarqubecloud bot commented Feb 10, 2026

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
97.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@JamesW1-NHS JamesW1-NHS temporarily deployed to internal-dev-sandbox February 10, 2026 11:13 — with GitHub Actions Inactive
@JamesW1-NHS JamesW1-NHS temporarily deployed to internal-dev-sandbox February 10, 2026 11:13 — with GitHub Actions Inactive
@JamesW1-NHS JamesW1-NHS merged commit 551b7b5 into master Feb 10, 2026
17 checks passed
@JamesW1-NHS JamesW1-NHS deleted the VED-1017-json-ack-file branch February 10, 2026 11:25
dlzhry2nhs added a commit that referenced this pull request Feb 17, 2026
@dlzhry2nhs @dependabot
@Akol125 @mfjarvis @JamesW1-NHS @edhall-nhs @FimranNHS
Feature/ved 000 merge latest from master (#1216)
1503884 
* VED-1059 Made adjustment to batch queueing logic (#1189)

* Bump @redocly/cli from 2.15.1 to 2.16.0 in the npm-minor-patch group (#1188)

Bumps the npm-minor-patch group with 1 update: [@redocly/cli](https://github.com/Redocly/redocly-cli).


Updates `@redocly/cli` from 2.15.1 to 2.16.0
- [Release notes](https://github.com/Redocly/redocly-cli/releases)
- [Commits](https://github.com/Redocly/redocly-cli/compare/@redocly/cli@2.15.1...@redocly/cli@2.16.0)

---
updated-dependencies:
- dependency-name: "@redocly/cli"
  dependency-version: 2.16.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-minor-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Daniel Yip <daniel.yip4@nhs.net>

* OAS documentation for name field characters (#1190)

* VED-901: Update extended attribute file destination. (#1141)

* VED-901: Update extended attributes destination prefix.

* VED-901: Update tests.

* VED-901: Upload EA files directly to the S3 submission upload bucket.

* VED-901: Lint tfvars files.

* VED-1017: Generate JSON ACK file (#1168)

* init

* log bugfix

* interim: create_json_ack_file. pending new unit test and fixing old ones

* TEMP_ACK_FILE

* update_json_ack_file

* ingestion_start_time

* complete_batch_file_process

* obtain_current_json_ack_content

* cut/paste fix

* mock out get_ingestion_start_time_by_message_id

* fix & test get_ingestion_start_time_by_message_id

* bugfix

* bugfix split()[]

* test_update_ack_file

* test_complete_batch_file_process_json_ack_file II

* code cleanup

* extra mocks

* supplier

* restored move_file

* json ack file unit tests for lambda_handler (part 1)

* archive source file test - to support e2e testing

* remove redundant functions

* bugfix: get message_id from correct place for JSON ack file

* Trigger build

* bug fix: get_ingestion_start_time_by_message_id

* fixed filename in JSON ack file

* fixed filename in JSON ack file II

* supplier

* unit tests

* cleanup

* kick pipeline

* remove duplicate update_audit_table_item() call

* remove duplicate update_audit_table_item() call II

* review changes

* success -> succeeded

* _make_json_ack_file_row

* refactor json_ack_data helper functions

* _generated_date test values

* update_csv_ack_file

* obtain_current_csv_ack_content

* _generated_date comment

* _make_json_ack_data_row

* manual merge of e2e_automation from #1177

* Remove grafana code, dependabot, README references and sonar config (#1193)

* Bump @redocly/cli from 2.16.0 to 2.17.0 in the npm-minor-patch group (#1192)

Bumps the npm-minor-patch group with 1 update: [@redocly/cli](https://github.com/Redocly/redocly-cli).


Updates `@redocly/cli` from 2.16.0 to 2.17.0
- [Release notes](https://github.com/Redocly/redocly-cli/releases)
- [Commits](https://github.com/Redocly/redocly-cli/compare/@redocly/cli@2.16.0...@redocly/cli@2.17.0)

---
updated-dependencies:
- dependency-name: "@redocly/cli"
  dependency-version: 2.17.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-minor-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Daniel Yip <daniel.yip4@nhs.net>

* VED-000 [NOJIRA] Update Python dependencies. (#1196)

* VED-1039 Added the new vaccine types permitted for TPP and Optum (EMIS) (#1200)

* Bump @redocly/cli from 2.17.0 to 2.18.0 in the npm-minor-patch group (#1198)

Bumps the npm-minor-patch group with 1 update: [@redocly/cli](https://github.com/Redocly/redocly-cli).


Updates `@redocly/cli` from 2.17.0 to 2.18.0
- [Release notes](https://github.com/Redocly/redocly-cli/releases)
- [Commits](https://github.com/Redocly/redocly-cli/compare/@redocly/cli@2.17.0...@redocly/cli@2.18.0)

---
updated-dependencies:
- dependency-name: "@redocly/cli"
  dependency-version: 2.18.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-minor-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Daniel Yip <daniel.yip4@nhs.net>

* Bump @redocly/cli from 2.18.0 to 2.18.1 in the npm-minor-patch group (#1208)

* Bump @redocly/cli from 2.18.0 to 2.18.1 in the npm-minor-patch group

Bumps the npm-minor-patch group with 1 update: [@redocly/cli](https://github.com/Redocly/redocly-cli).


Updates `@redocly/cli` from 2.18.0 to 2.18.1
- [Release notes](https://github.com/Redocly/redocly-cli/releases)
- [Commits](https://github.com/Redocly/redocly-cli/compare/@redocly/cli@2.18.0...@redocly/cli@2.18.1)

---
updated-dependencies:
- dependency-name: "@redocly/cli"
  dependency-version: 2.18.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-minor-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Trigger build

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Daniel Yip <daniel.yip4@nhs.net>

* Bump the terraform-minor-patch group across 2 directories with 1 update (#1209)

Bumps the terraform-minor-patch group with 1 update in the /infrastructure/account directory: [hashicorp/aws](https://github.com/hashicorp/terraform-provider-aws).
Bumps the terraform-minor-patch group with 1 update in the /infrastructure/instance directory: [hashicorp/aws](https://github.com/hashicorp/terraform-provider-aws).


Updates `hashicorp/aws` from 6.31.0 to 6.32.1
- [Release notes](https://github.com/hashicorp/terraform-provider-aws/releases)
- [Changelog](https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md)
- [Commits](hashicorp/terraform-provider-aws@v6.31.0...v6.32.1)

Updates `hashicorp/aws` from 6.31.0 to 6.32.1
- [Release notes](https://github.com/hashicorp/terraform-provider-aws/releases)
- [Changelog](https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md)
- [Commits](hashicorp/terraform-provider-aws@v6.31.0...v6.32.1)

Updates `hashicorp/aws` from 6.31.0 to 6.32.1
- [Release notes](https://github.com/hashicorp/terraform-provider-aws/releases)
- [Changelog](https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md)
- [Commits](hashicorp/terraform-provider-aws@v6.31.0...v6.32.1)

Updates `hashicorp/aws` from 6.31.0 to 6.32.1
- [Release notes](https://github.com/hashicorp/terraform-provider-aws/releases)
- [Changelog](https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md)
- [Commits](hashicorp/terraform-provider-aws@v6.31.0...v6.32.1)

---
updated-dependencies:
- dependency-name: hashicorp/aws
  dependency-version: 6.32.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: terraform-minor-patch
- dependency-name: hashicorp/aws
  dependency-version: 6.32.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: terraform-minor-patch
- dependency-name: hashicorp/aws
  dependency-version: 6.32.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: terraform-minor-patch
- dependency-name: hashicorp/aws
  dependency-version: 6.32.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: terraform-minor-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Daniel Yip <daniel.yip4@nhs.net>

* VED-1038 updated lint issue (#1207)

* updated the smoke tag and add scenarios as required in ticket

* ruff format

* fix the broken tests

* updated lint issue

* Update tests/e2e_automation/features/APITests/update.feature

Co-authored-by: Daniel Yip <daniel.yip4@nhs.net>

* fix broken test by adding the valid vaccination in pre-condition

* fix lint issue

---------

Co-authored-by: Daniel Yip <daniel.yip4@nhs.net>

* VED-981 Add base infrastructure for the MNS Publisher feature (#1191)

* VED-981 (Part 2) Add DLQ, redrive policy, and encryption to queues (#1203)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Akinola Olutola <akinola.olutola1@nhs.net>
Co-authored-by: Matt Jarvis <61870561+mfjarvis@users.noreply.github.com>
Co-authored-by: James W. <218585646+JamesW1-NHS@users.noreply.github.com>
Co-authored-by: Ed Hall <239591530+edhall-nhs@users.noreply.github.com>
Co-authored-by: FimranNHS <fatima.imran3@nhs.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dlzhry2nhs dlzhry2nhs dlzhry2nhs approved these changes

@Akol125 Akol125 Awaiting requested review from Akol125

@edhall-nhs edhall-nhs Awaiting requested review from edhall-nhs

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JamesW1-NHS @dlzhry2nhs @edhall-nhs

Comments